Vector selection decision for pixel interpolation

ABSTRACT

Methods and apparatus, including computer program products, implementing and using techniques for determining motion vectors to be used for interpolation of intermediary frames in a digital video sequence are disclosed. A first image frame including several image patches is received. A second image frame including one or more image patches corresponding to the image patches in the first image frame is received. For each image patch that occurs in both frames, the following operations occur: forward and backward motion vectors are determined for the image patch in the first image frame, forward and backward motion vectors are determined for the image patch in the second image frame, one motion vector from the first image frame and one motion vector from the second frame are selected and the selected motion vectors are used to establish an intermediary position of the image patch in an interpolated frame between the frames.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims benefit of priority from U.S. Provisional PatentApplication No. 60/532,427, filed Dec. 23, 2003, and entitled “RecursiveHierarchical Motion Compensated Frame Rate Conversion,” which is herebyincorporated by reference herein. This application is also related toU.S. patent application entitled: “Motion Vector Computation For VideoSequences” Ser. No. 10/832,838 by Nair et al., filed Apr. 26, 2004,which is hereby incorporated by reference herein in its entirety.

BACKGROUND

This invention relates to improving video and graphics processing.

At low display refresh rates (for example, 50 fields/sec for interlacedvideo material, and 24 frames/sec for film-originated material) ondigital display devices, a display artifact referred to as “areaflicker” can occur. The area flicker becomes more visible as the size ofthe display increases, due to the high sensitivity to flicker in thehuman visual peripheral region. A simple solution for reducing the areaflicker is to increase the display refresh rate by repeating the inputfields or frames at a higher rate (for example, 100 fields/sec forinterlaced video). This solves the area flicker problem for staticscenes. However, the repetition introduces a new artifact in scenes withmotion, known as “motion judder” or “motion smear,” particularly inareas with high contrast, due to the human eye's tendency to track thetrajectory of moving objects. For this reason, motion compensated frameinterpolation is preferred, in which the pixels are computed in aninterpolated frame or field at an intermediate point on a local motiontrajectory, so that there is no discrepancy between an expected imagemotion due to eye tracking and a displayed image motion. A motion vectordescribes the local image motion trajectory from one field or frame tothe next.

Motion vectors can be computed at different levels of spatialresolution, such as at a pixel level, at an image patch level, or at anobject level. Computing a motion vector for every pixel independentlywould theoretically result in an ideal data set, but is unfeasible dueto the large number of computations required. Computing a motion vectorfor each image patch reduces the number of computations, but can resultin artifacts due to motion vector discontinuities within an image patch.Computing motion vectors on an object basis can theoretically result inhigh resolution and lower computational requirements, but objectsegmentation is a challenging problem.

Therefore what is needed is a way to determine and use motion vectorsefficiently and accurately, such that little or no discrepancy existsbetween an expected image motion due to eye tracking and a displayedimage motion in a digital video.

SUMMARY

The present invention provides methods and apparatus for determining andusing motion vectors efficiently and accurately, such that little or nodiscrepancy exists between an expected image motion due to eye trackingand a displayed image motion in a digital video.

In general, in one aspect, the invention provides methods and apparatus,including computer program products, implementing and using techniquesfor determining motion vectors to be used for interpolation ofintermediary frames in a digital video sequence. A first image frameincluding several image patches is received. Each image patch has arespective first position. A second image frame including one or moreimage patches corresponding to the image patches in the first imageframe is received. Each image patch has a respective second position.For each image patch in the first image frame that has a correspondingimage patch in the second frame, the following operations occur: aforward motion vector and a backward motion vector is determined for theimage patch in the first image frame; a forward motion vector and abackward motion vector is determined for the image patch in the secondimage frame; a pair of motion vectors consisting of one motion vectorfrom the first image frame and one motion vector from the second frameis selected; and the selected pair of motion vectors is used toestablish an intermediary position of the image patch in an interpolatedframe that is intermediary to the first and second frames.

Advantageous implementations can include one or more of the followingfeatures. Selecting a pair of motion vectors can include calculating aweight for one or more pairs of motion vectors consisting of one motionvector from the first image frame and one motion vector from the secondframe, and selecting the pair of motion vectors having the lowestweight. Calculating a weight can include calculating an absolutedifference of an x-component of the motion vector from the first imageframe and an x-component of the motion vector from the second imageframe, calculating an absolute difference of an y-component of themotion vector from the first image frame and an y-component of themotion vector from the second image frame, adding the calculatedabsolute differences for the x- and y-components, and multiplying theadded calculated absolute differences with a correlation factor for themotion vector from the first image frame and the motion vector from thesecond image frame.

Multiplying can include calculating a correlation value for the motionvector from the first image frame, calculating a correlation value forthe motion vector from the second image frame, adding the two calculatedcorrelation values, and multiplying the added correlation values withthe added calculated absolute differences of the motion vectorcomponents from the motion vector from the first image frame and themotion vector from the second image frame. Calculating a correlationvalue for the motion vector from the first image frame can includecentering a first window on a pixel in the first image frame that formsan origin of the motion vector, centering a second window on a pixel inthe second image frame that forms an end point of the motion vector, thesecond window having the same dimensions as the first window; andcalculating a sum of absolute differences of luma values for the pixelsin the first window and pixels at corresponding positions in the secondwindow. The dimensions of the first and second windows can be identicalto the dimensions of the image patch.

The pair of motion vectors can consist of one of the followingcombinations: a forward motion vector from the first image frame and aforward vector from the second image frame, a backward motion vectorfrom the first image frame and a backward vector from the second imageframe, and a forward motion vector from the first image frame and abackward vector from the second image frame. Selecting can includeselecting any one pair of motion vectors if the image patch is part of alarge textured object moving with constant velocity. Selecting caninclude selecting the forward motion vector from the first image frameand the backward motion vector from the second image frame if the imagepatch is part of an accelerating object. Selecting can include selectingthe forward motion vector from the first image frame and the forwardmotion vector from the second image frame if the image patch is part ofan area being uncovered by a trailing edge of a moving object. Selectingcan include selecting the backward motion vector from the first imageframe and the backward motion vector from the second image frame if theimage patch is part of an area being covered by a leading edge of amoving object. Each image patch can include many pixels.

The details of one or more embodiments of the invention are set forth inthe accompanying drawings and the description below. Other features,objects, and advantages of the invention will be apparent from thedescription and drawings, and from the claims.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 shows a flowchart of a recursive hierarchical process fordetermining a motion vector.

FIG. 2 shows a series of image frames in a digital video of a movingobject, and the areas where forward and backward motion vectors can bedetermined.

FIG. 3 shows a flowchart of a method for using a pair of motion vectorsto interpolate a location of an image patch in an image frameintermediate to two original image frames in a digital video.

FIG. 4 is a schematic view of a sequence of image frames and associatedmotion vectors.

FIG. 5 shows a computer system employed to implement the invention.

Like reference symbols in the various drawings indicate like elements.

DETAILED DESCRIPTION OF THE INVENTION

The present invention provides methods and apparatus for determiningmotion vectors and using determined motion vectors efficiently andaccurately, such that little or no discrepancy exists between anexpected image motion due to eye tracking and a displayed image motionin a digital video. The motion vectors that are used are selected suchthat the number of visual artifacts is minimized.

Determining Motion Vectors

First, an exemplary approach of determining motion vectors will bediscussed. The motion vector determination described herein uses arecursive hierarchical approach, which has been fully described in theabove referenced patent application “Motion Vector Computation for VideoSequences,” which is incorporated herein by reference in its entirety.However, it should be realized that regardless of the approach that ischosen for determining motion vectors, the motion vectors can be usedfor interpolation as described below in the section “Using the motionvectors.”

Generally, for motion compensated approaches to work well, including therecursive hierarchical approach described herein, two basic assumptionsare made about the nature of the object motion: 1) moving objects haveinertia, and 2) moving objects are large. The inertia assumption impliesthat a motion vector changes only gradually with respect to a temporalvector sampling interval (that is, the frame rate in the digital video).The large objects assumption implies that a motion vector changes onlygradually with respect to a spatial vector sampling interval, that is,the vector field is smooth and has only few boundary motiondiscontinuities.

The goal of the recursive hierarchical method is to find a motion vectorby applying a source correlation window to a first image frame and atarget correlation window to a subsequent image frame, and placing thetarget correlation window such that a best match with the sourcecorrelation window is obtained, that is, the contents of the sourcecorrelation window and target correlation window are as similar aspossible. At the same time, the number of calculations needed to performthe matching between the source correlation window and the targetcorrelation window must be as low as possible, while still searching theentire vector space limit. In order to accomplish these goals, therecursive hierarchical method uses multiple resolution levels of theimage frames. A best motion vector is first determined for the lowestresolution level by projecting the previous best motion vector at thehighest resolution level down to the lowest resolution level, andtesting it and one or more updates. This best motion vector is thenpropagated up to a higher resolution level, where some adjustments aremade and a new best motion vector is determined. This new best motionvector is propagated up to yet another higher resolution level, wheremore adjustments are made and another new best motion vector isdetermined. This process is repeated until the highest, original,resolution level has been reached and a best motion vector has beenidentified.

FIG. 1 shows one implementation of a recursive hierarchical process(100). It is assumed that multiple resolution levels of the image frameshave already been generated. As can be seen in FIG. 1, the recursivehierarchical process (100) for determining a motion vector starts byprojecting a motion vector from a previous image frame down to a lowestresolution level (step 102). A set of update vectors is generated andtested to find a best motion vector at this lowest resolution level(step 104). In one implementation this test is performed by comparingpixels in corresponding positions in a source correlation windowcentered on the origin of the motion vector and a target correlationwindow centered on the end point of each respective update vector. Thecomparison can, for example, be performed by subtracting a luma valuefor each pixel in the source window from the corresponding pixel in therespective target windows. In this case the best match would be definedby finding a minimum sum of absolute differences (SAD) for a sourcecorrelation window and a target correlation window pair, and the bestmotion vector would be the vector associated with this sourcecorrelation window and a target correlation window pair.

In one implementation, the SAD is computed by letting the candidatevectors for an image patch, which all originate at the same image patchlocation in the source frame, point to different pixel locations in atarget frame. For each candidate vector, a rectangular window iscentered in the target frame on the pixel pointed to by the respectivecandidate vector. A corresponding rectangular window is centered in thesource frame on the pixel where the candidate vectors originate. Then apair-wise absolute difference of the corresponding luma pixels in thetwo windows, that is, the pixels that have the same relative locationwithin the two windows, is calculated. The sum of all the absolutedifferences is the SAD value. The SAD decreases as the window matchingbecomes better and is ideally zero when the pixels are identical. Inpractice, of course, due to noise and other factors, the best vectorwill have a non-zero SAD, but will have the minimum SAD of the vectorsin the set of candidate vectors.

After the minimum SAD has been found, the best vector is selected (step106). The process (100) then examines whether there are any higherresolution levels (step 108). If there are higher resolution levels, theprocess propagates the best vector up to the next higher resolutionlevel (step 110) and repeats steps 104 through 108. If there are nohigher resolution levels, the process proceeds to step 112, where thebest vector is selected as the motion vector and is used for motioncompensation, which completes the process for the current frame.

In one implementation, a camera vector is also considered whencalculating a best motion vector. The camera vector describes a globalmovement of the contents of the frame, as opposed to the local vectorsat each image patch location that are computed completely independently,and can therefore be used to aid in finding a better true motion vector.In several commonly occurring scenarios a motion vector resulting fromcamera movements at every location in a frame can be predicted quiteeasily with a simple model. For example, in the case of a camera lenspanning across a distant landscape, all the motion vectors will beidentical and equivalent to the velocity of the camera. Another scenariois when a camera lens zooms into an object on a flat surface, such as apicture on a wall. All the motion vectors then have a radial directionand increase from zero at the image center to a maximum value at theimage periphery.

In one implementation, the process tries to fit a mathematical model tothe motion vectors that have been computed using a least squares method.A good fit between the camera motion vectors and the mathematical modelindicates that one of the scenarios discussed above likely is present,and the camera model predicted vector can then be used as an additionalcandidate vector in the next recursive hierarchical vector estimationstep. Taking the camera vector into consideration is advantageous inthat the recursive portion of the recursive hierarchical search is alocal search approach, which may converge into a false local minimuminstead of the true minimum. The camera predicted vector candidate canpotentially help in avoiding detection of false local minima and directthe process towards a true minimum

The main advantage of the recursive hierarchical approach is that at alower level, an update of a pixel is equivalent to an update of two ormore pixels at the next higher level, depending on the difference inresolution between the two levels. If there are, for example, threeresolution levels, say 1:1, 1:2 and 1:4, and an update of ±/−1 pixel ateach level, the convergence delay is potentially reduced by a factor offour. Expressed differently, effectively the resolution hierarchy isused to accelerate the temporal recursion convergence. This results insignificant improvements, in particular for frames containing smallobjects moving with high velocities.

As can be seen from the above discussion, a smooth and accurate vectorfield is provided by using only a fairly small number of calculations.Furthermore, there is reduced convergence delay due to the multiplelevels of resolution. Fewer resolution levels can be used compared toconventional approaches, and vector errors in lower levels are notamplified due to resolution changes at higher resolution levels due tosafeguarding by use of projected vectors at each resolution.

Using the Motion Vectors

After the motion vector fields have been estimated for each frame,interpolated frames in between the original frames of the digital videocan be computed. The robustness of the interpolation can be increased byusing motion vectors from more than one of the original frames.Conventional approaches to generate interpolated frames includegenerating data values from the candidate vectors followed by afiltering scheme to reduce the visibility of artifacts due to erroneousvectors.

The present invention instead uses a heuristic algorithm to select thebest possible candidates from the available set of motion vectors, whichfurther increases the robustness and decreases the number of artifacts.The heuristic algorithm will be discussed in detail below with referenceto FIG. 3, and is applicable to situations where there aresimultaneously available forward and backward vectors from two adjacentoriginal frames. The heuristic scheme uses both the vector correlationvalues and the distance between candidate vectors to select a pair offorward vectors, a pair of backward vectors, or a pair consisting of aforward and a backward vector. Before discussing the heuristic algorithmin detail, it is useful to discuss some properties of forward andbackward vectors

The above discussion has been focused on determining motion vectorsbetween a current frame and a subsequent frame. These motion vectors aretypically referred to as forward motion vectors. However, just asforward motion vectors can be determined, the same methodology can beused to determine a motion vector between a current frame and a previousframe. Since this type of motion vector points to a previous frame, itis often referred to as a backward motion vector. Thus, a pair ofvectors consisting of one motion vector from a previous frame and onemotion vector from a subsequent frame can be used to determine thelocation of an image patch in an intermediate interpolated frame.

It should be noted that there are situations in which it is impossibleto determine a forward and a backward motion vector for each of twoadjacent frames. One such example is shown in FIG. 2, which shows asimplified one-dimensional object (200) moving along a line upwards inFIG. 2, against a static background in four adjacent image frames (202A-202 D) in a digital video. Here it is assumed that a location of theobject (200) in an interpolated image frame (212) is to be determinedbetween the original frames 202 B and 202 C, using one motion vectorfrom frame 202 B and one motion vector from frame 202 C.

As can be seen in FIG. 2, frame 202 B contains a background region (204)that was previously covered by the object (200) in frame 202 A.Therefore, a backward vector cannot be computed for this backgroundregion (204) in frame 202 B. Similarly, frame 202 C contains abackground region (206) that was previously covered by the object (200)in frame 202 B, and consequently a backward vector cannot be computedfor this background region (206) in frame 202 C. In other words, forareas that are being uncovered by the trailing edge of a moving object,no backward motion vectors can be determined.

Considering the leading edge of the object (200), it can be seen thatframe 202 B contains a background region (208) that will be obscured bythe leading edge of the object (200) in the following frame 202 C.Therefore, a forward vector cannot be computed for this backgroundregion (208) in frame 202 B. Similarly, frame 202 C contains abackground region (210) that will be covered by the object (200) inframe 202 D, and therefore a forward vector cannot be computed for thisbackground region (210) in frame 202 C. Stated differently, for areasthat are being covered by the leading edge of a moving object in asubsequent frame, no forward motion vectors can be determined. Thereader skilled in the art will realize that these two situations arejust exemplary, and that other situations, for example, an object movingtowards or away from the camera, may occur that make it impossible todetermine both forward and backward motion vectors in both adjacentframes. In this situation, the resultant size change of the object fromone image frame to a subsequent image frame will cause either visiblebackground to be obscured, or obscured background to be uncovered. Thus,an object moving towards the camera will cause trouble with computationof forward vectors at the object's boundary, whereas an object movingaway from the camera will cause trouble with computation of backwardvectors at the object's boundary.

FIG. 3 shows a heuristic motion vector selection process (300) that canbe used to choose a pair of motion vectors. FIG. 4 shows a sequence(400) of image frames (402; 406, 408; 410), and will be used inconjunction with FIG. 3 to explain the steps of the heuristic motionselection process (300). As can be seen, the process (300) starts bydetermining motion vectors for two frames (404, 406) that surround aframe (405) that will be interpolated (step 302). These two frames willbe referred to as the previous frame (404) and the subsequent frame(406) in relation to the interpolated frame (405).

The motion vectors can be determined with the recursive hierarchicalmethod described above, or with any other suitable method fordetermining motion vectors. For each frame, the process (300) attemptsto determine a forward motion vector (414, 418) and a backward motionvector (412, 416). In the following discussion, the forward motionvector of the previous frame (404) will be labeled Fp (414), thebackward motion vector of the previous frame (404) will be labeled Bp(412), the forward motion vector of the subsequent frame (406) will belabeled Fs (418), and the backward motion vector of the subsequent frame(406) will be labeled Bs (416).

Next, the process (300) checks whether there are both forward motionvectors (414, 418) and backward motion vectors (412, 416) available forboth the previous frame (404) and the subsequent frame (406) (step 304).If all vectors are not available, the process (300) ends and uses onlyone motion vector. If all vectors are available, the process (300)selects a pair of vectors to be used for the interpolation of theintermediate frame. As the skilled reader will realize, there are fourpossible combinations of vector pairs that may be generated from thedetermined forward vectors (414, 418) and backward vectors (412, 416).These combinations are:

-   -   1. Fp (414) and Fs (418)    -   2. Bp (412) and Bs (416)    -   3. Fp (414) and Bs (416)    -   4. Bp (412) and Fs (418)

However, only the first three vector pairs make sense to consider, sinceboth the motion vectors Bp (412) and Fs (418) in the last pair both“point away” from the frame that is to be interpolated, that is, bothvectors refer to movement that occurs outside the time interval forwhich the interpolated frame is to be computed.

Furthermore, different vector pairs among the three remaining vectorpairs are more or less suitable in different situations. In the eventthat the type of object motion can be independently determined, it willoften be apparent what vector pair to use when determining theinterpolated frame. For example, in the case of a large textured objectmoving with constant velocity, any one pair of the first three pairs ofmotion vectors on the above list can be used to generate consistentframe interpolated data, since the object moves the same distancebetween each image frame.

For an accelerating object, on the other hand, the Fp (414) and Fs (418)vectors are different. For example, if the object (410) is acceleratingin the downward direction of FIG. 4, the Fp vector (414) will have asmaller magnitude than the Fs vector (418). Similarly, the Bp (412) andBs (416) vectors are different. If the same acceleration is assumed, theBp vector (412) will have a smaller magnitude than the Bs vector (416).However, the Fp vector (414) and the Bs vector (416) are computed overthe same frame interval—the Fp vector (414) has the opposite directionand magnitude of the Bs vector (416)—and can therefore be used for theinterpolation.

Yet another example is the one discussed above with respect to FIG. 2,of a background area being uncovered at the trailing edge of a movingobject. The forward vectors Fp and Fs can always be determined for abackground area that is being uncovered at the trailing edge of a movingobject, but the backward vectors Bp and Bs can never be determined,since the background area was covered by the object previous frames. Forthis reason, the (Fp, Fs) vector pair is the only vector pair that canbe used for the interpolation.

Similarly, in the case of a background area being obscured at theleading edge of a moving object, the (Bp, Bs) vector pair is the onlypair that can be used for the interpolation. No forward vectors can bedetermined, since the leading edge of the moving object will obscure thebackground area in subsequent image frames.

If the type of object movement is not known, which is typically thecase, the heuristical process continues by determining which vector pairshould be used. The process calculates a weight (W1 through W3) for eachof the first three motion vector pairs as follows (step 306):W1=dist(Fp,Fs)*(corr(Fp)+corr(Fs))W2=dist(Bp,Bs)*(corr(Bp)+corr(Bs))W3=dist(Fp,−Bs)*(corr(Fp)+corr(Bs))where dist (A,B) is the distance between vectors A and B, and iscalculated as the sum of the absolute differences of the x components ofvectors A and B, and the absolute differences of the y components ofvectors A and B, respectively. The corr (A) measure is the SADcorrelation value for the vector. As was discussed above, the SAD islower as the local image match increases, that is, as the vectorconfidence increases.

Next, the process selects the vector pair that gives the lowestdetermined weight (step 308), since this pair provides the highestconsistency. Finally, the selected vector pair is used for final datainterpolation (step 310). This heuristic vector selection processincreases the robustness of the frame interpolation and minimizesartifacts, which are often manifested as a “shimmering” effect whenviewed on a screen, which would occur if only one motion vector was usedin the interpolation.

The invention can be implemented in digital electronic circuitry, or incomputer hardware, firmware, software, or in combinations of them.Apparatus of the invention can be implemented in a computer programproduct tangibly embodied in a machine-readable storage device forexecution by a programmable processor; and method steps of the inventioncan be performed by a programmable processor executing a program ofinstructions to perform functions of the invention by operating on inputdata and generating output. The invention can be implementedadvantageously in one or more computer programs that are executable on aprogrammable system including at least one programmable processorcoupled to receive data and instructions from, and to transmit data andinstructions to, a data storage system, at least one input device, andat least one output device. Each computer program can be implemented ina high-level procedural or object-oriented programming language, or inassembly or machine language if desired; and in any case, the languagecan be a compiled or interpreted language. Suitable processors include,by way of example, both general and special purpose microprocessors.Generally, a processor will receive instructions and data from aread-only memory and/or a random access memory. Generally, a computerwill include one or more mass storage devices for storing data files;such devices include magnetic disks, such as internal hard disks andremovable disks; magneto-optical disks; and optical disks. Storagedevices suitable for tangibly embodying computer program instructionsand data include all forms of non-volatile memory, including by way ofexample semiconductor memory devices, such as EPROM, EEPROM, and flashmemory devices; magnetic disks such as internal hard disks and removabledisks; magneto-optical disks; and CD-ROM disks. Any of the foregoing canbe supplemented by, or incorporated in, ASICs (application-specificintegrated circuits).

FIG. 5 shows a computer system (500) employed to implement theinvention. The computer system (500) is only an example of a graphicssystem in which the present invention can be implemented. The computersystem (500) includes a central processing unit (CPU) (510), a randomaccess memory (RAM) (520), a read only memory (ROM) (525), one or moreperipherals (530), a graphics controller (560), primary storage devices(540 and 550), and a digital display unit (570). As is well known in theart, ROM acts to transfer data and instructions uni-directionally to theCPUs (510), while the RAM (520) is used typically to transfer data andinstructions in a bi-directional manner. The CPUs (510) can generallyinclude any number of processors. Both primary storage devices (540 and550) can include any suitable computer-readable media. A secondarystorage medium (580), which is typically a mass memory device, is alsocoupled bi-directionally to the CPUs (510) and provides additional datastorage capacity. The mass memory device (580) is a computer-readablemedium that can be used to store programs including computer code, data,and the like. Typically, the mass memory device (580) is a storagemedium such as a hard disk or a tape which generally slower than theprimary storage devices (540, 550). The mass memory storage device (580)can take the form of a magnetic or paper tape reader or some otherwell-known device. It will be appreciated that the information retainedwithin the mass memory device (580), can, in appropriate cases, beincorporated in standard fashion as part of the RAM (520) as virtualmemory.

The CPUs (510) are also coupled to one or more input/output devices(590) that can include, but are not limited to, devices such as videomonitors, track balls, mice, keyboards, microphones, touch-sensitivedisplays, transducer card readers, magnetic or paper tape readers,tablets, styluses, voice or handwriting recognizers, or other well-knowninput devices such as, of course, other computers. Finally, the CPUs(510) optionally can be coupled to a computer or telecommunicationsnetwork, e.g., an Internet network or an intranet network, using anetwork connection as shown generally at (595). With such a networkconnection, it is contemplated that the CPUs (510) might receiveinformation from the network, or might output information to the networkin the course of performing the above-described method steps. Theabove-described devices and materials will be familiar to those of skillin the computer hardware and software arts.

The graphics controller (560) generates image data and a correspondingreference signal, and provides both to digital display unit (570). Theimage data can be generated, for example, based on pixel data receivedfrom the CPU (510) or from an external encode (not shown). In oneembodiment, the image data is provided in RGB format and the referencesignal includes the VSYNC and HSYNC signals well known in the art.However, it should be understood that the present invention can beimplemented with data and/or reference signals in other formats.

A number of implementations of the invention have been described.Nevertheless, it will be understood that various modifications can bemade without departing from the spirit and scope of the invention. Forexample in addition to the hierarchical and temporal vectors in theintermediate layers, the camera model generated vector projected downcan also be used as a candidate for SAD computation. Furthermore, themotion vectors generated as described above can be used for otherpurposes than frame rate conversion, such as deinterlacing, noisereduction, and so on. The robustness can also be increased further by,for example, using motion vectors determined for frames that are notimmediately adjacent, but instead located two frames away on either sideof the interpolated frame in the digital video. Accordingly, otherembodiments are within the scope of the following claims.

1. A method for determining motion vectors to be used for interpolationof intermediary frames in a digital video sequence, comprising:receiving a first image frame including a plurality of image patches,each image patch having a respective first position; receiving a secondimage frame including one or more image patches corresponding to theimage patches in the first image frame, each image patch having arespective second position; for each image patch in the first imageframe that has a corresponding image patch in the second frame:determining a forward motion vector and a backward motion vector for theimage patch in the first image frame; determining a forward motionvector and a backward motion vector for the image patch in the secondimage frame; selecting a pair of motion vectors consisting of one motionvector from the first image frame and one motion vector from the secondframe; and using the selected pair of motion vectors to establish anintermediary position of the image patch in an interpolated frame thatis intermediary to the first and second frames.
 2. The method of claim1, wherein selecting a pair of motion vectors comprises: calculating aweight for one or more pairs of motion vectors consisting of one motionvector from the first image frame and one motion vector from the secondframe; and selecting the pair of motion vectors having the lowestweight.
 3. The method of claim 2, wherein calculating a weightcomprises: calculating an absolute difference of an x-component of themotion vector from the first image frame and an x-component of themotion vector from the second image frame; calculating an absolutedifference of an y-component of the motion vector from the first imageframe and an y-component of the motion vector from the second imageframe; adding the calculated absolute differences for the x- andy-components; and multiplying the added calculated absolute differenceswith a correlation factor for the motion vector from the first imageframe and the motion vector from the second image frame.
 4. The methodof claim 3, wherein multiplying comprises: calculating a correlationvalue for the motion vector from the first image frame; calculating acorrelation value for the motion vector from the second image frame;adding the two calculated correlation values; and multiplying the addedcorrelation values with the added calculated absolute differences of thecomponents of the motion vector from the first image frame and themotion vector from the second image frame.
 5. The method of claim 4,wherein calculating a correlation value for the motion vector from thefirst image frame comprises: centering a first window on a pixel in thefirst image frame that forms an origin of the motion vector; centering asecond window on a pixel in the second image frame that forms an endpoint of the motion vector, the second window having the same dimensionsas the first window; and calculating a sum of absolute differences ofluma values for the pixels in the first window and pixels atcorresponding positions in the second window.
 6. The method of claim 5,wherein the dimensions of the first and second windows are identical tothe dimensions of the image patch.
 7. The method of claim 1, wherein thepair of motion vectors consists of one of the following combinations: aforward motion vector from the first image frame and a forward vectorfrom the second image frame, a backward motion vector from the firstimage frame and a backward vector from the second image frame, and aforward motion vector from the first image frame and a backward vectorfrom the second image frame.
 8. The method of claim 1, wherein selectingcomprises: selecting any one pair of motion vectors if the image patchis part of a large textured object moving with constant velocity.
 9. Themethod of claim 1, wherein selecting comprises: selecting the forwardmotion vector from the first image frame and the backward motion vectorfrom the second image frame if the image patch is part of anaccelerating object.
 10. The method of claim 1, wherein selectingcomprises: selecting the forward motion vector from the first imageframe and the forward motion vector from the second image frame if theimage patch is part of an area being uncovered by a trailing edge of amoving object.
 11. The method of claim 1, wherein selecting comprises:selecting the backward motion vector from the first image frame and thebackward motion vector from the second image frame if the image patch ispart of an area being covered by a leading edge of a moving object. 12.The method of claim 1, wherein each image patch includes a plurality ofpixels.
 13. A computer readable medium encoded with a computer programhaving computer executable instructions when executed by a computerenables the computer to determine motion vectors to be used forinterpolation of intermediary frames in a digital video sequence, thecomputer-executable instructions comprising: receive a first image frameincluding a plurality of image patches, each image patch having arespective first position; receive a second image frame including one ormore image patches corresponding to the image patches in the first imageframe, each image patch having a respective second position; for eachimage patch in the first image frame that has a corresponding imagepatch in the second frame: determine a forward motion vector and abackward motion vector for the image patch in the first image frame;determine a forward motion vector and a backward motion vector for theimage patch in the second image frame; select a pair of motion vectorsconsisting of one motion vector from the first image frame and onemotion vector from the second frame; and use the selected pair of motionvectors to establish an intermediary position of the image patch in aninterpolated frame that is intermediary to the first and second frames.14. The computer readable medium of claim 13, wherein instructions toselect a pair of motion vectors comprise instructions to: calculate aweight for one or more pairs of motion vectors consisting of one motionvector from the first image frame and one motion vector from the secondframe; and select the pair of motion vectors having the lowest weight.15. The computer readable medium of claim 14 wherein the instructions tocalculate a weight comprise instructions to: calculate an absolutedifference of an x-component of the motion vector from the first imageframe and an x-component of the motion vector from the second imageframe; calculate an absolute difference of an y-component of the motionvector from the first image frame and an y-component of the motionvector from the second image frame; add the calculated absolutedifferences for the x- and y-components; and multiply the addedcalculated absolute differences with a correlation factor for the motionvector from the first image frame and the motion vector from the secondimage frame.
 16. The computer readable medium of claim 15 wherein theinstructions to multiply comprise instructions to: calculate acorrelation value for the motion vector from the first image frame;calculate a correlation value for the motion vector from the secondimage frame; add the two calculated correlation values; and multiply theadded correlation values with the added calculated absolute differencesof the components of the motion vector from the first image frame andthe motion vector from the second image frame.
 17. The computer readablemedium of claim 16 wherein the a correlation value for the motion vectorfrom the first image frame comprise instructions to: center a firstwindow on a pixel in the first image frame that forms an origin of themotion vector; center a second window on a pixel in the second imageframe that forms an end point of the motion vector, the second windowhaving the same dimensions as the first window; and calculate a sum ofabsolute differences of luma values for the pixels in the first windowand pixels at corresponding positions in the second window.
 18. Thecomputer readable medium of claim 17, wherein the dimensions of thefirst and second windows are identical to the dimensions of the imagepatch.
 19. The computer readable medium of claim 13, wherein the pair ofmotion vectors consists of one of the following combinations: a forwardmotion vector from the first image frame and a forward vector from thesecond image frame, a backward motion vector from the first image frameand a backward vector from the second image frame, and a forward motionvector from the first image frame and a backward vector from the secondimage frame.
 20. The computer readable medium of claim 13, wherein theinstructions to select comprise instructions to: select any one pair ofmotion vectors if the image patch is part of a large textured objectmoving with constant velocity.
 21. The computer readable medium of claim13, wherein the instructions to select comprise instructions to: selectthe forward motion vector from the first image frame and the backwardmotion vector from the second image frame if the image patch is part ofan accelerating object.
 22. The computer readable medium of claim 13wherein the instructions to select comprise instructions to: select theforward motion vector from the first image frame and the forward motionvector from the second image frame if the image patch is part of an areabeing uncovered by a trailing edge of a moving object.
 23. The computerreadable medium of claim 13, wherein the instructions to select compriseinstructions to: select the backward motion vector from the first imageframe and the backward motion vector from the second image frame if theimage patch is part of an area being covered by a leading edge of amoving object.
 24. The computer readable medium of claim 13, whereineach image patch includes a plurality of pixels.